[Open.ogc] Deleted layers still exist?

John Kennedy - NOAA Affiliate john.f.kennedy at noaa.gov
Tue Sep 24 20:14:15 UTC 2013 

Interesting question. I did a preview and the features do appear to be
overwritten in the layer. But the "new" attributes are not loading as the
column names defer between the old and the new shape file. I checked a
number of polygons and got blank values as shown below.

[image: Inline image 1]

John F. Kennedy, PhD, CPG
 GIS Specialist
Contractor: Caelum Corp.
NOAA Fisheries Office of Science and Technology
1315 East-West Highway
SSMC3 Rm 9518
Silver Spring, MD 20910
301-427-8149

Comments? Have I been of assistance? Please let my supervisors know. :~)
Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov>
Hussain Jabalpurwala - NOAA Affiliate <hussain.jabalpurwala at noaa.gov>


On Tue, Sep 24, 2013 at 4:05 PM, Micah Wengren <micah.wengren at noaa.gov>wrote:

>  Sure, no problem.  So if you take your new shapefile with modified
> columns and try to upload it to the original layer name 'communities_pgs',
> can you tell for sure that it does not overwrite data in the columns that
> haven't changed from the old version to the new?  It should be doing that
> at least, and discarding any new columns, as well as probably leaving any
> original columns that were removed in the new file empty.  At least that
> would be better than not overwriting anything at all.
>
> Micah
>
>
>
> On 9/24/2013 3:47 PM, John Kennedy - NOAA Affiliate wrote:
>
> If the name of the shape file is the same as one of the layers, then
> geoserver presents the old data. I took the shape file I'm attempting to
> load as communities_pgs and named it communities_pgs20130924 and it loads
> as expected with the new columns and data.
>
>  Thanks for all your help on this issue.
>
>  John F. Kennedy, PhD, CPG
>  GIS Specialist
>  Contractor: Caelum Corp.
> NOAA Fisheries Office of Science and Technology
> 1315 East-West Highway
> SSMC3 Rm 9518
> Silver Spring, MD 20910
> 301-427-8149
>
>  Comments? Have I been of assistance? Please let my supervisors know. :~)
> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov>
>  Hussain Jabalpurwala - NOAA Affiliate <hussain.jabalpurwala at noaa.gov>
>
>
> On Tue, Sep 24, 2013 at 3:32 PM, Micah Wengren <micah.wengren at noaa.gov>wrote:
>
>>  John,
>>
>> You may have hit on an issue with the way our system works.  Can you tell
>> if the attribute values that are being saved in the database are coming
>> from the new shapefile you just tried to upload?  Could you insert a test
>> value and try the upload again?  I just want to check that the new data is
>> actually being written when you specify update=overwrite.  It should be,
>> but it would be good to confirm.
>>
>> I don't actually know how GeoServer handles the overwrite option behind
>> the scenes.  I thought it might drop the underlying database table and
>> recreate, but it may not be.  That would be why the schema is being
>> preserved.  Best way to find out is for me to open a ticket with OpenGeo.
>> I'll do that later today.
>>
>> Micah
>>
>>
>> On 9/24/2013 2:59 PM, John Kennedy - NOAA Affiliate wrote:
>>
>> Okay, I now feel a little silly. I did not replace the dummy password in
>> my notes file with my LDAP password. Shape files and zip files have the
>> same names as the layer. I attempted the data load again, using --update
>> overwrite, and I got a 201 response. However, when I look at the layer in
>> geoserver I see the old data. There is a column (property) called
>> GEOID_jlf and that column is in the old data layer and not in the new
>> layer I'm attempting to load.
>>
>>  [image: Inline image 1]
>>
>>
>>  John F. Kennedy, PhD, CPG
>>  GIS Specialist
>>  Contractor: Caelum Corp.
>> NOAA Fisheries Office of Science and Technology
>> 1315 East-West Highway
>> SSMC3 Rm 9518
>> Silver Spring, MD 20910
>> 301-427-8149
>>
>>  Comments? Have I been of assistance? Please let my supervisors know. :~)
>> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov>
>>  Hussain Jabalpurwala - NOAA Affiliate <hussain.jabalpurwala at noaa.gov>
>>
>>
>> On Tue, Sep 24, 2013 at 2:35 PM, Micah Wengren <micah.wengren at noaa.gov>wrote:
>>
>>>  Ok.  Just curious, what are the shapefile names you're trying to
>>> upload?  Are they the communities_pts and communities_pgs layers?
>>>
>>> Looks like those are gone from the FeatureTypes list now (
>>> https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis.html).
>>> But they still show up in the list when you go to the 'New Layer' page.
>>> Not sure I've seen that issue before, so might have to file a ticket.
>>>
>>> As for credentials, it should only be your NOAA LDAP username and
>>> password.  Those don't get passed on to the database, there's a separate
>>> way database authorization is handled.  If you can log in to the Web UI,
>>> there shouldn't be an issue with the REST API, or at least you shouldn't be
>>> getting that 401 response.
>>>
>>> Micah
>>>
>>>
>>> On 9/24/2013 2:21 PM, John Kennedy - NOAA Affiliate wrote:
>>>
>>> Thanks. Tomorrow works better for me. As for the credentials, nothing as
>>> changed on this end. I was able to load data with the hpcc_data.py script
>>> up to yesterday, but I did get an error message when attempting to
>>> overwrite the layer/feature type. Now, there maybe credential issues in the
>>> database (PostGIS?) but I've no way of knowing.
>>>
>>>  John F. Kennedy, PhD, CPG
>>>  GIS Specialist
>>>  Contractor: Caelum Corp.
>>> NOAA Fisheries Office of Science and Technology
>>> 1315 East-West Highway
>>> SSMC3 Rm 9518
>>> Silver Spring, MD 20910
>>> 301-427-8149
>>>
>>>  Comments? Have I been of assistance? Please let my supervisors know.
>>> :~)
>>> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov>
>>>  Hussain Jabalpurwala - NOAA Affiliate <hussain.jabalpurwala at noaa.gov>
>>>
>>>
>>> On Tue, Sep 24, 2013 at 2:15 PM, Micah Wengren <micah.wengren at noaa.gov>wrote:
>>>
>>>>  Hi John,
>>>>
>>>> That is odd for the 401 response on the data upload.  Are you sure the
>>>> credentials are right?  The other two seem ok, depending on if the
>>>> featuretype existed in the first place or not.
>>>>
>>>> I might be able to come by later, but right now I have a deadline to
>>>> finish something this afternoon.  Let me know if you aren't able to get it
>>>> to work.
>>>>
>>>> Micah
>>>>
>>>>
>>>> On 9/24/2013 12:32 PM, John Kennedy - NOAA Affiliate wrote:
>>>>
>>>> Hi Micah,
>>>>
>>>>  Well, I think I'm getting close. I installed pip, python setup, and
>>>> request. I used hpcc_date_requests.py to delete the layer and the feature
>>>> type and I get these to responses:
>>>> Response: 200, url=
>>>> https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis/featuretypes/communities_pgs
>>>>
>>>>  Response: 404, url=
>>>> https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis/featuretypes/communities_pgs
>>>>
>>>>  The hpcc_date_requests appears to be attempting to load data, but the
>>>> response back is this:
>>>> Response: 401, url=
>>>> https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis/file.shp?update=overwrite&configure=all
>>>>
>>>>  Any chance you can come to my cube and have a look?
>>>>
>>>>  Cheers,
>>>> John
>>>>
>>>>
>>>>
>>>>  John F. Kennedy, PhD, CPG
>>>>  GIS Specialist
>>>>  Contractor: Caelum Corp.
>>>> NOAA Fisheries Office of Science and Technology
>>>> 1315 East-West Highway
>>>> SSMC3 Rm 9518
>>>> Silver Spring, MD 20910
>>>> 301-427-8149
>>>>
>>>>  Comments? Have I been of assistance? Please let my supervisors know.
>>>> :~)
>>>> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov>
>>>>  Hussain Jabalpurwala - NOAA Affiliate <hussain.jabalpurwala at noaa.gov>
>>>>
>>>>
>>>> On Tue, Sep 24, 2013 at 10:18 AM, Micah Wengren <micah.wengren at noaa.gov
>>>> > wrote:
>>>>
>>>>>  Hi John,
>>>>>
>>>>> How are you doing the attempted data upload?  With the python script?
>>>>> Can you tell me which python script you are using if so?  If you can, try
>>>>> using the hpcc_data_requests.py script instead of the other one (might
>>>>> involve some admin rights to install python modules on your workstation).
>>>>> I've update that one more recently.  I'd like to drop the other one at some
>>>>> point for easier maintenance.
>>>>>
>>>>> I think you should be able to upload a new data set to replace the
>>>>> existing one, even if it already exists in the database and data store (but
>>>>> could be wrong if there's an issue going on here with these layers).
>>>>> Something like the following should work:
>>>>>
>>>>> > hpcc_data_requests.py -a data -u <name> -p <pass> -w nmfs_st -f
>>>>> /path/to/shapefile.zip --type shape --update overwrite  (the shapefile
>>>>> would need to be named to match the layer you want to update).
>>>>>
>>>>>
>>>>> Also, this is something that is a little whacky with the GeoServer
>>>>> REST API (and should be better documented somewhere admittedly), but you
>>>>> may need to do a delete command for both the Layer and the FeatureType to
>>>>> fully delete it.  When you create a new layer with the Web UI, it actually
>>>>> makes both the Layer and FeatureType behind the scenes.  Not sure if this
>>>>> will fix this particular issue, but commands to do that would be something
>>>>> like:
>>>>>
>>>>> > hpcc_data_requests.py -a delete_layer -u <name> -p <pass> -w nmfs_st
>>>>> --layername <layername>
>>>>> > hpcc_data_requests.py -a delete_featuretype -u <name> -p <pass> -w
>>>>> nmfs_st --layername <layername>
>>>>>
>>>>> This should fully delete both a layer and its corresponding
>>>>> featuretype.  It's possible if you did a delete_layer command with via the
>>>>> script without doing delete_featuretype as well, some featuretypes got left
>>>>> behind.  Again, an issue that should really be better documented in
>>>>> GeoServer docs but unfortunately I haven't seen it described anywhere, I
>>>>> happened to find out myself after having some similar issues.
>>>>>
>>>>> I'm looking around the workspace now, and I think I see some cases
>>>>> where there are FeatureTypes that aren't associated with any layers, in
>>>>> particular 'communities_pgs' and 'communities_pts' are both FeatureTypes
>>>>> without corresponding layers.  This could be preventing the data upload
>>>>> from happening for these layers.  I'm just looking at these two lists:
>>>>>
>>>>>
>>>>> https://services.ogc.noaa.gov/geoserver/web/?wicket:bookmarkablePage=:org.geoserver.web.data.layer.LayerPage(filtered by 'nmfs_st')
>>>>>
>>>>> https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis.html
>>>>>
>>>>> Also, you should probably do the delete_featuretype command for
>>>>> 'communities_pgs20130506' and 'communities_pts20130506', since they don't
>>>>> have corresponding Layers.  I think that will get those two in sync.
>>>>>
>>>>> I'm not sure why you get such a long list in the New Layer dialog
>>>>> though, looks like a bug to me.  No idea where 'SI_SE_Gulf_sites_pg_ALLsites'
>>>>> and 'SI_SE_Gulf_sites_pt' are coming from, because those aren't
>>>>> listed as current FeatureTypes in the REST API.
>>>>>
>>>>> HTH,
>>>>> Micah
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 9/24/2013 9:30 AM, John Kennedy - NOAA Affiliate wrote:
>>>>>
>>>>> The two layers of concern are communities_pgs and communities_pts.
>>>>> When I attempt to load new data, via a zip file with a shape file, I get a
>>>>> message that the communities_pgs exists. When I look at the list of source
>>>>> under the "Add sources" button I can see the communities_pgs layer and the
>>>>> publish link. But when I click the link and attempt to save I get the
>>>>> message below. Deleting layer via the hpcc_data python script also seems to
>>>>> fail.
>>>>>
>>>>>  [image: Inline image 1]
>>>>>
>>>>>  John F. Kennedy, PhD, CPG
>>>>>  GIS Specialist
>>>>>  Contractor: Caelum Corp.
>>>>> NOAA Fisheries Office of Science and Technology
>>>>> 1315 East-West Highway
>>>>> SSMC3 Rm 9518
>>>>> Silver Spring, MD 20910
>>>>> 301-427-8149
>>>>>
>>>>>  Comments? Have I been of assistance? Please let my supervisors know.
>>>>> :~)
>>>>> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov>
>>>>>  Hussain Jabalpurwala - NOAA Affiliate <hussain.jabalpurwala at noaa.gov>
>>>>>
>>>>>
>>>>> On Tue, Sep 24, 2013 at 9:20 AM, Tim Haverland - NOAA Federal <
>>>>> tim.haverland at noaa.gov> wrote:
>>>>>
>>>>>> I'm adding John Kennedy to this thread so he can work with you on
>>>>>> this issue.
>>>>>>
>>>>>>  Tim
>>>>>>
>>>>>>
>>>>>> On Tue, Sep 24, 2013 at 8:49 AM, Micah Wengren <
>>>>>> micah.wengren at noaa.gov> wrote:
>>>>>>
>>>>>>>  Tim,
>>>>>>>
>>>>>>> I can see some layers in your workspace in the New Layer dialog that
>>>>>>> aren't currently published and have a 'Publish' link next to them, so I see
>>>>>>> what you mean.  Are you able to create a new SQL view layer with the same
>>>>>>> name as one of those layers, or does that not work?  Can you give me the
>>>>>>> name of one of the layers that you've tried with and I'll pass it on to
>>>>>>> OpenGeo support to troubleshoot, that way I can give them a specific layer
>>>>>>> name with the issue.
>>>>>>>
>>>>>>> The layers should be disappearing but for whatever reason they're
>>>>>>> not.
>>>>>>>
>>>>>>> Micah
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 9/23/2013 3:55 PM, Tim Haverland - NOAA Federal wrote:
>>>>>>>
>>>>>>>  Hi all,
>>>>>>>
>>>>>>>  We've been trying to clean up a bunch of layers in our workspace.
>>>>>>> We can delete layers using the "Remove selected resources" button on the
>>>>>>> Layers page, but when we try to replace that layer with another, the layer
>>>>>>> is still there.
>>>>>>>
>>>>>>>  We can verify this by clicking "Add a new resource" ... all of the
>>>>>>> old layers are still there and I can click Publish to make them "live"
>>>>>>> again.
>>>>>>>
>>>>>>>  How do we permanently get rid of unwanted layers?
>>>>>>>
>>>>>>>  Tim
>>>>>>> --
>>>>>>> *Tim Haverland*
>>>>>>> Acting Operations Branch Chief
>>>>>>> NOAA Fisheries Office of Science and Technology
>>>>>>> 1315 East-West Highway
>>>>>>> SSMC3 Rm 12303
>>>>>>> Silver Spring, MD 20910
>>>>>>> 301-427-8137
>>>>>>>
>>>>>>>
>>>>>>>  _______________________________________________
>>>>>>> Open.ogc mailing listOpen.ogc at list.woc.noaa.govhttps://list.woc.noaa.gov/cgi-bin/mailman/listinfo/open.ogc
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>  --
>>>>>> *Tim Haverland*
>>>>>> Acting Operations Branch Chief
>>>>>> NOAA Fisheries Office of Science and Technology
>>>>>> 1315 East-West Highway
>>>>>> SSMC3 Rm 12303
>>>>>> Silver Spring, MD 20910
>>>>>> 301-427-8137
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://list.woc.noaa.gov/pipermail/open.ogc/attachments/20130924/93bc8ba5/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 164721 bytes
Desc: not available
URL: <https://list.woc.noaa.gov/pipermail/open.ogc/attachments/20130924/93bc8ba5/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 166250 bytes
Desc: not available
URL: <https://list.woc.noaa.gov/pipermail/open.ogc/attachments/20130924/93bc8ba5/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 153411 bytes
Desc: not available
URL: <https://list.woc.noaa.gov/pipermail/open.ogc/attachments/20130924/93bc8ba5/attachment-0005.png>

More information about the Open.ogc mailing list