[Open.ogc] Deleted layers still exist?

Tue Sep 24 19:47:09 UTC 2013

If the name of the shape file is the same as one of the layers, then
geoserver presents the old data. I took the shape file I'm attempting to
load as communities_pgs and named it communities_pgs20130924 and it loads
as expected with the new columns and data.

Thanks for all your help on this issue.

John F. Kennedy, PhD, CPG
 GIS Specialist
Contractor: Caelum Corp.
NOAA Fisheries Office of Science and Technology
1315 East-West Highway
SSMC3 Rm 9518
Silver Spring, MD 20910
301-427-8149

Comments? Have I been of assistance? Please let my supervisors know. :~)
Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov>
Hussain Jabalpurwala - NOAA Affiliate <hussain.jabalpurwala at noaa.gov>

On Tue, Sep 24, 2013 at 3:32 PM, Micah Wengren <micah.wengren at noaa.gov>wrote:

>  John,
>
> You may have hit on an issue with the way our system works.  Can you tell
> if the attribute values that are being saved in the database are coming
> from the new shapefile you just tried to upload?  Could you insert a test
> value and try the upload again?  I just want to check that the new data is
> actually being written when you specify update=overwrite.  It should be,
> but it would be good to confirm.
>
> I don't actually know how GeoServer handles the overwrite option behind
> the scenes.  I thought it might drop the underlying database table and
> recreate, but it may not be.  That would be why the schema is being
> preserved.  Best way to find out is for me to open a ticket with OpenGeo.
> I'll do that later today.
>
> Micah
>
>
> On 9/24/2013 2:59 PM, John Kennedy - NOAA Affiliate wrote:
>
> Okay, I now feel a little silly. I did not replace the dummy password in
> my notes file with my LDAP password. Shape files and zip files have the
> same names as the layer. I attempted the data load again, using --update
> overwrite, and I got a 201 response. However, when I look at the layer in
> geoserver I see the old data. There is a column (property) called
> GEOID_jlf and that column is in the old data layer and not in the new
> layer I'm attempting to load.
>
>  [image: Inline image 1]
>
>
>  John F. Kennedy, PhD, CPG
>  GIS Specialist
>  Contractor: Caelum Corp.
> NOAA Fisheries Office of Science and Technology
> 1315 East-West Highway
> SSMC3 Rm 9518
> Silver Spring, MD 20910
> 301-427-8149
>
>  Comments? Have I been of assistance? Please let my supervisors know. :~)
> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov>
>  Hussain Jabalpurwala - NOAA Affiliate <hussain.jabalpurwala at noaa.gov>
>
>
> On Tue, Sep 24, 2013 at 2:35 PM, Micah Wengren <micah.wengren at noaa.gov>wrote:
>
>>  Ok.  Just curious, what are the shapefile names you're trying to
>> upload?  Are they the communities_pts and communities_pgs layers?
>>
>> Looks like those are gone from the FeatureTypes list now (
>> https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis.html).
>> But they still show up in the list when you go to the 'New Layer' page.
>> Not sure I've seen that issue before, so might have to file a ticket.
>>
>> As for credentials, it should only be your NOAA LDAP username and
>> password.  Those don't get passed on to the database, there's a separate
>> way database authorization is handled.  If you can log in to the Web UI,
>> there shouldn't be an issue with the REST API, or at least you shouldn't be
>> getting that 401 response.
>>
>> Micah
>>
>>
>> On 9/24/2013 2:21 PM, John Kennedy - NOAA Affiliate wrote:
>>
>> Thanks. Tomorrow works better for me. As for the credentials, nothing as
>> changed on this end. I was able to load data with the hpcc_data.py script
>> up to yesterday, but I did get an error message when attempting to
>> overwrite the layer/feature type. Now, there maybe credential issues in the
>> database (PostGIS?) but I've no way of knowing.
>>
>>  John F. Kennedy, PhD, CPG
>>  GIS Specialist
>>  Contractor: Caelum Corp.
>> NOAA Fisheries Office of Science and Technology
>> 1315 East-West Highway
>> SSMC3 Rm 9518
>> Silver Spring, MD 20910
>> 301-427-8149
>>
>>  Comments? Have I been of assistance? Please let my supervisors know. :~)
>> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov>
>>  Hussain Jabalpurwala - NOAA Affiliate <hussain.jabalpurwala at noaa.gov>
>>
>>
>> On Tue, Sep 24, 2013 at 2:15 PM, Micah Wengren <micah.wengren at noaa.gov>wrote:
>>
>>>  Hi John,
>>>
>>> That is odd for the 401 response on the data upload.  Are you sure the
>>> credentials are right?  The other two seem ok, depending on if the
>>> featuretype existed in the first place or not.
>>>
>>> I might be able to come by later, but right now I have a deadline to
>>> finish something this afternoon.  Let me know if you aren't able to get it
>>> to work.
>>>
>>> Micah
>>>
>>>
>>> On 9/24/2013 12:32 PM, John Kennedy - NOAA Affiliate wrote:
>>>
>>> Hi Micah,
>>>
>>>  Well, I think I'm getting close. I installed pip, python setup, and
>>> request. I used hpcc_date_requests.py to delete the layer and the feature
>>> type and I get these to responses:
>>> Response: 200, url=
>>> https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis/featuretypes/communities_pgs
>>>
>>>  Response: 404, url=
>>> https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis/featuretypes/communities_pgs
>>>
>>>  The hpcc_date_requests appears to be attempting to load data, but the
>>> response back is this:
>>> Response: 401, url=
>>> https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis/file.shp?update=overwrite&configure=all
>>>
>>>  Any chance you can come to my cube and have a look?
>>>
>>>  Cheers,
>>> John
>>>
>>>
>>>
>>>  John F. Kennedy, PhD, CPG
>>>  GIS Specialist
>>>  Contractor: Caelum Corp.
>>> NOAA Fisheries Office of Science and Technology
>>> 1315 East-West Highway
>>> SSMC3 Rm 9518
>>> Silver Spring, MD 20910
>>> 301-427-8149
>>>
>>>  Comments? Have I been of assistance? Please let my supervisors know.
>>> :~)
>>> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov>
>>>  Hussain Jabalpurwala - NOAA Affiliate <hussain.jabalpurwala at noaa.gov>
>>>
>>>
>>> On Tue, Sep 24, 2013 at 10:18 AM, Micah Wengren <micah.wengren at noaa.gov>wrote:
>>>
>>>>  Hi John,
>>>>
>>>> How are you doing the attempted data upload?  With the python script?
>>>> Can you tell me which python script you are using if so?  If you can, try
>>>> using the hpcc_data_requests.py script instead of the other one (might
>>>> involve some admin rights to install python modules on your workstation).
>>>> I've update that one more recently.  I'd like to drop the other one at some
>>>> point for easier maintenance.
>>>>
>>>> I think you should be able to upload a new data set to replace the
>>>> existing one, even if it already exists in the database and data store (but
>>>> could be wrong if there's an issue going on here with these layers).
>>>> Something like the following should work:
>>>>
>>>> > hpcc_data_requests.py -a data -u <name> -p <pass> -w nmfs_st -f
>>>> /path/to/shapefile.zip --type shape --update overwrite  (the shapefile
>>>> would need to be named to match the layer you want to update).
>>>>
>>>>
>>>> Also, this is something that is a little whacky with the GeoServer REST
>>>> API (and should be better documented somewhere admittedly), but you may
>>>> need to do a delete command for both the Layer and the FeatureType to fully
>>>> delete it.  When you create a new layer with the Web UI, it actually makes
>>>> both the Layer and FeatureType behind the scenes.  Not sure if this will
>>>> fix this particular issue, but commands to do that would be something like:
>>>>
>>>> > hpcc_data_requests.py -a delete_layer -u <name> -p <pass> -w nmfs_st
>>>> --layername <layername>
>>>> > hpcc_data_requests.py -a delete_featuretype -u <name> -p <pass> -w
>>>> nmfs_st --layername <layername>
>>>>
>>>> This should fully delete both a layer and its corresponding
>>>> featuretype.  It's possible if you did a delete_layer command with via the
>>>> script without doing delete_featuretype as well, some featuretypes got left
>>>> behind.  Again, an issue that should really be better documented in
>>>> GeoServer docs but unfortunately I haven't seen it described anywhere, I
>>>> happened to find out myself after having some similar issues.
>>>>
>>>> I'm looking around the workspace now, and I think I see some cases
>>>> where there are FeatureTypes that aren't associated with any layers, in
>>>> particular 'communities_pgs' and 'communities_pts' are both FeatureTypes
>>>> without corresponding layers.  This could be preventing the data upload
>>>> from happening for these layers.  I'm just looking at these two lists:
>>>>
>>>>
>>>> https://services.ogc.noaa.gov/geoserver/web/?wicket:bookmarkablePage=:org.geoserver.web.data.layer.LayerPage(filtered by 'nmfs_st')
>>>>
>>>> https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis.html
>>>>
>>>> Also, you should probably do the delete_featuretype command for
>>>> 'communities_pgs20130506' and 'communities_pts20130506', since they don't
>>>> have corresponding Layers.  I think that will get those two in sync.
>>>>
>>>> I'm not sure why you get such a long list in the New Layer dialog
>>>> though, looks like a bug to me.  No idea where 'SI_SE_Gulf_sites_pg_ALLsites'
>>>> and 'SI_SE_Gulf_sites_pt' are coming from, because those aren't listed
>>>> as current FeatureTypes in the REST API.
>>>>
>>>> HTH,
>>>> Micah
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 9/24/2013 9:30 AM, John Kennedy - NOAA Affiliate wrote:
>>>>
>>>> The two layers of concern are communities_pgs and communities_pts. When
>>>> I attempt to load new data, via a zip file with a shape file, I get a
>>>> message that the communities_pgs exists. When I look at the list of source
>>>> under the "Add sources" button I can see the communities_pgs layer and the
>>>> publish link. But when I click the link and attempt to save I get the
>>>> message below. Deleting layer via the hpcc_data python script also seems to
>>>> fail.
>>>>
>>>>  [image: Inline image 1]
>>>>
>>>>  John F. Kennedy, PhD, CPG
>>>>  GIS Specialist
>>>>  Contractor: Caelum Corp.
>>>> NOAA Fisheries Office of Science and Technology
>>>> 1315 East-West Highway
>>>> SSMC3 Rm 9518
>>>> Silver Spring, MD 20910
>>>> 301-427-8149
>>>>
>>>>  Comments? Have I been of assistance? Please let my supervisors know.
>>>> :~)
>>>> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov>
>>>>  Hussain Jabalpurwala - NOAA Affiliate <hussain.jabalpurwala at noaa.gov>
>>>>
>>>>
>>>> On Tue, Sep 24, 2013 at 9:20 AM, Tim Haverland - NOAA Federal <
>>>> tim.haverland at noaa.gov> wrote:
>>>>
>>>>> I'm adding John Kennedy to this thread so he can work with you on this
>>>>> issue.
>>>>>
>>>>>  Tim
>>>>>
>>>>>
>>>>> On Tue, Sep 24, 2013 at 8:49 AM, Micah Wengren <micah.wengren at noaa.gov
>>>>> > wrote:
>>>>>
>>>>>>  Tim,
>>>>>>
>>>>>> I can see some layers in your workspace in the New Layer dialog that
>>>>>> aren't currently published and have a 'Publish' link next to them, so I see
>>>>>> what you mean.  Are you able to create a new SQL view layer with the same
>>>>>> name as one of those layers, or does that not work?  Can you give me the
>>>>>> name of one of the layers that you've tried with and I'll pass it on to
>>>>>> OpenGeo support to troubleshoot, that way I can give them a specific layer
>>>>>> name with the issue.
>>>>>>
>>>>>> The layers should be disappearing but for whatever reason they're
>>>>>> not.
>>>>>>
>>>>>> Micah
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 9/23/2013 3:55 PM, Tim Haverland - NOAA Federal wrote:
>>>>>>
>>>>>>  Hi all,
>>>>>>
>>>>>>  We've been trying to clean up a bunch of layers in our workspace.
>>>>>> We can delete layers using the "Remove selected resources" button on the
>>>>>> Layers page, but when we try to replace that layer with another, the layer
>>>>>> is still there.
>>>>>>
>>>>>>  We can verify this by clicking "Add a new resource" ... all of the
>>>>>> old layers are still there and I can click Publish to make them "live"
>>>>>> again.
>>>>>>
>>>>>>  How do we permanently get rid of unwanted layers?
>>>>>>
>>>>>>  Tim
>>>>>> --
>>>>>> *Tim Haverland*
>>>>>> Acting Operations Branch Chief
>>>>>> NOAA Fisheries Office of Science and Technology
>>>>>> 1315 East-West Highway
>>>>>> SSMC3 Rm 12303
>>>>>> Silver Spring, MD 20910
>>>>>> 301-427-8137
>>>>>>
>>>>>>
>>>>>>  _______________________________________________
>>>>>> Open.ogc mailing listOpen.ogc at list.woc.noaa.govhttps://list.woc.noaa.gov/cgi-bin/mailman/listinfo/open.ogc
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>  --
>>>>> *Tim Haverland*
>>>>> Acting Operations Branch Chief
>>>>> NOAA Fisheries Office of Science and Technology
>>>>> 1315 East-West Highway
>>>>> SSMC3 Rm 12303
>>>>> Silver Spring, MD 20910
>>>>> 301-427-8137
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://list.woc.noaa.gov/pipermail/open.ogc/attachments/20130924/9154ba5c/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 153411 bytes
Desc: not available
URL: <https://list.woc.noaa.gov/pipermail/open.ogc/attachments/20130924/9154ba5c/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 164721 bytes
Desc: not available
URL: <https://list.woc.noaa.gov/pipermail/open.ogc/attachments/20130924/9154ba5c/attachment-0003.png>